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A thorough solution to the problem of determining the error rate of a 
digital communication system with intersymbol interference and additive 
Gaussian noise is presented in this paper. The solution achieves for the 
first time a combination of computational simplicity and a high degree 
of accuracy, and is obtained by deriving tight upper and lower bounds 
on the error rate. It is shown that, for a system with a normalized peak 
distortion less than unity, these bounds can be made to differ by an arbi- 
trarily small amount. The numerical evaluation of the bounds takes less 
than one second on the GE-Mark II time-sharing system for almost all 
the cases. 

Examples are given for 2M-ary digital systems to demonstrate the 
accuracy and computational efficiency of our method. The results show 
that our estimates of error rate are generally orders of magnitude better 
than the Chemoff bound. For example, in the case of an ideal bandlimited 
system [(sin t)/t pulse shape] with a signal-to-noise ratio of 16 dB and a 
sampling instant deviation of 0.05 from the optimum value, the lower 
and upper bounds on the error rate are 1.1 X 10~ s and 1.2 X 10'*, respec- 
tively. 

This method can also be applied to the calculation of the performance 
of certain phase-shift-keyed systems and certain sijstems with co-channel 
interference. 

I. INTRODUCTION 

In many cases the transmission efficiency of a digital system is 
largely limited by intersymbol interference rather than by additive 
noise. Intersymbol interference may result from imperfect design of 
the filters, distortion in the transmission channel, nonideal sampling 
instant, or nonideal demodulating carrier phase. In analyzing such a 
digital data system, it is important to determine the system error rate 
due to intersymbol interference and additive noise. 
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Various methods' ~ 7 to evaluate the error rate have been proposed. 
They provide either a loose upper bound of the error rate or the error 
rate of a channel with truncated impulse response. 

In this paper we present a simple method to evaluate both an upper 
and a lower bound of the error rate without invoking the finite pulse- 
train approximation. Furthermore, it is shown that for a system with 
a normalized peak distortion less than unity, the upper and lower 
bounds can be made arbitrarily close thus obtaining an accurate estimate 
of the error rate of the system. This method can be applied to 2M-ary 
AM and coherent phase-shift-keyed systems. 

The data system model will be described briefly in Section II. Various 
proposed techniques to evaluate the error probability and their draw- 
backs are discussed in Section III. In Section IV, we will present new 
upper and lower bounds and the computation of the bounds by a series 
expansion. Applications and the convergence properties of the bounds 
are described in Section V. Throughout, additive Gaussian noise and 
independence of information digits are assumed. 

II. BRIEF DESCRIPTION OF THE SYSTEM 

A simplified block diagram of a digital AM data system is shown in 
Fig. 1. We assume that an impulse 8(t) having amplitude a t is trans- 
mitted through the channel every T seconds. The system transfer 
function is 

R{u) = S(a>)!T( W ).E(co). (1) 

In the absence of channel noise, a sequence of input signals, 

Z a t 8(t - IT), (2) 

will generate a corresponding output sequence, 

Z a t r{t - IT), (3) 

where r(t) is the Fourier transform of R(co), {a t } is a sequence of in- 
dependent random variables, and a, = ±1, ±3, • • • ± (2M — 1) 
with equal probability for all integers, I. We also assume that additive 
Gaussian noise is present in the system. Thus the corrupted received 
sequence at the input to the receiver detector is 

V(t) = t, a,r(t - IT) + n(t), (4) 
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Fig. 1 — Simplified block diagram of a 2Af-ary data system. 



where n(t) is additive Gaussian noise with power a 2 watts. At the 
detector, y(t) is sampled every T seconds to determine the amplitude 
of the transmitted signal. At sampling time t , the sampled signal is 



y(to) = a r(t ) + 2 M<o ~ IT) + n(t ). 



(5) 



l = -e 



The first term is the desired signal while the second and the third 
terms represent the intersymbol interference and the Gaussian noise 
respectively. 

The set of slicing levels is 1 

0, =fc2r(« ), ±4'-(/ ), ••■ , ±(2m - 2)r(t ). (6) 

Based on the decision levels given by equation (6), for a particular 
transmitted signal level, a„ , the conditional error probability is 

\P[y(to) ^ -^(m - l)r(/ )}, a = -(2m - 1) 

P{y(t ) ^ 2(m - l)r(to)}, a„ = 2m - 1 

P{[y(t ) ^ (a„ + l)r(to)]U[y(to) ^ (<*> ~ MOD, 

a„ ^ ±(2m - 1), 

where A UB is the union of the events A and B. 
Substituting equation (5) into (7), we obtain 

P{ £ a,r(* - IT) + n(t„) ^ r(t )}, a = -(2m - 1) 



P r (e/a ) = « 



(7) 



P r (e/a ) = - 



P{ T. a,r(t n - IT) + n(t ) ^ -r(t )}, a = 2m - 1 



(8) 



/>{[£ a,r(/„ - /r) + n(<„) ^ r(* )]E/ 
[£ a,r(«a - ?T) + »(0 ^ -KOD, a ^ ±(2m - 1). 
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Since £*"„ a t r(t - IT) and n(t ) are equally likely to be positive or 
negative, equation (8) reduces to 

\P[ E a t r(tc - IT) + n(to) ^ r(U)}, a Q = ±(2ro - 1) 

(9) 
2P{ £ a t r(t - IT) + n(t ) ^ r(Q}, a n * ±(2m - 1). 



P r (e/a ) = < 



1*0 



The error rate of the system is 
P< = E Pr(e/ao)P r (a ) 

= [(2m - l)/m]P{ 2 «'K'o - W) + n(* ) ^ r(/ )} • (10) 

We notice that in equation (10) the variables m, a, , and n(t ) have 
already been denned. The sequence r(t — IT) is assumed to be known* 
in the following sense: 

r(t — IT) is finite and known Viz S N , (11) 

where S^ is a set of N + 1 distinct integers (including I = 0) and f 

Z r 2 (t - IT) = d < oo . (12) 

Define 

X = 2> ( r(* - W). (13) 

From equation (12) we conclude that the infinite sum X converges 
absolutely to a random variable and equation (10) can be alternately 
written as 

P. f (27TCT 2 )- 1 f exp {-[y - r(0 + X] 2 /2<r 2 } rfy dF(Z). (14) 

•'all A' J-oo 

III. REVIEW OF EXISTING METHODS 

The existing methods of evaluating equation (10) can be divided 
into the following categories. 

3.1 Worst Case Estimate 

A worst case sequence 1 or "eye pattern" analysis is frequently used 
to analyze a data system. The error probability is estimated by setting 

* The sequence r(t — IT) is either experimentally determined or calculated 
through the system transfer function. 

t ff r 2 is obtained through the application of Parseval's theorem to equivalent 
Nyquist pulse (p. 47, Ref. 1). 
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^2i *o 0-iI'Hm — IT) to its worst case value in equation (10). In many 
cases, this estimate is exceedingly pessimistic since the occurrence of 
such a worst case sequence is extremely rare. 

3.2. Chernoff Bound 

Recently, Saltzberg 2 and Lugannani 3 applied the Chebyshev in- 
equality to equation (10) to obtain the upper bound on error prob- 
ability. We have shown in Ref. 6 that these upper bounds are in many 
cases still too pessimistic by orders of magnitude. 

3.3. Finite Truncated Pulse Train Approximation 4 ' 5 

When r(t) decreases rapidly relative to the sampling period T, we 
may approximate the channel by a finitely truncated pulse train. 
The error rate can be calculated by enumerating all the possible com- 
binations of intersymbol interference. However, since each calculation 
of the conditional error probability takes a great deal of computer 
time, the number of m N must be held to several thousand. 1 This limita- 
tion leads to a poor approximation of the true channel, and the error 
probability so obtained is not very useful. Recently, Hill 8 has reported 
that by computer simulation of the density function of X lV , the com- 
putation time can be reduced. 

3.4 Series Expansion Method 

Recently, Ho and Yeh° and, independently, Shimbo and Celebiler 7 
discovered that equation (10) can be calculated in terms of an ab- 
solutely convergent series involving moments of the intersymbol 
interference.* Furthermore, the moments can be obtained readily 
through recurrence relations, and the computation time is greatly 
reduced. A better approximation of the real channel can be obtained 
by increasing the number of terms in the pulse train approximation. 
However, the error in the P f estimate introduced by the truncation 
of the system impulse response is still unknown. 

IV. ERROR BOUNDS AND COMPUTATION TECHNIQUES 

In this section we shall derive new upper and lower bounds on the 
error rates and define the range of applicability of our method. No 
truncation of the intersymbol interference is required. Furthermore, 
this method will give an accurate estimate of the error rate with a 
negligible amount of computation time. 



* Only truncated pulse train approximations are considered. 
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4.1 Upper and Lower Bound of P e 

Let the intersymbol interference be partitioned into two disjoint 
sets where 

X N = £ a t r(t n - IT), (15a) 

1*0 
ItSN 

and 

X R = X a ir (t - IT). (15b) 

ItSN 

Equation (14) can be rewritten as 
P e = [(2m - l)/m] f f (2x(7 2 )-' 

Jail Xn ''all Xr 

■ f exp \-[y - r(t ) + X N + X R ] 2 /2a 2 } dy dF(X N ) dF(X R ). 

J -00 

(16) 
Proposition 1 : P e is lower bounded by 

P L = [(2m - l)/m] ( (W)~* 

t* al 1 Zj 

• f exp [- {?/ - K4) + ^I72a a ] dy dF(X N ), (17) 

J-oo 

provided the truncated system has an "open eye pattern," i.e., 

KQ - Z \r(to~ IT) | ^ 0. (18) 



lc.S.v 
1*0 



Proof: The complementary error function is concave upwards for 
negative values of its argument and satisfies the following relationship: 

\ erfc (z + a) + § erfc (z - a) ^ erfc (z), z ^ 0. (19) 

Since X R is symmetrically distributed around zero and X N satisfies 
equation (18), we obtain, by applying equation (19), that 

f f exp \-[y- r(t ) + X* + X R ] 2 /2a 2 } dy dF(X R ) 

•'all Xr J-oo 

> f exp {-[y- r(t ) + X„] 2 /2<r 2 } cty. (20) 

Substituting equation (20) into equation (16), we obtain the lower 
bound of equation (17). 
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Proposition 2: P e is upper bounded by 
P u = [(2m - l)/m] f (27r(r 2 )" 5 

Jail A'.v 

■f exp {-[y - r(to) + X m ?/?tf) dydF(XJ, (21a) 

where 

<r 2 = v\\ - wl/ST\ (21b) 

al = (l/3)(2m- l)(2m + l)<r*, (21c) 

and a 2 T is defined in equation (12). 
Prooj: Applying the following inequality, 

exp{-X fl 2 /2a 2 } ^ 1, (22) 

to equation (16), we obtain 

P e g [(2m - l)/m] f f f (2WY h 

J *\\ Xn J -so ''all Xr 

.exp{-[y-r(t ) + X N ) 2 /2* 2 } 

■ exp { - [y - r(t ) + X N ]X R /a 2 } dF{X R ) dy dF(XJ. (23) 
Knowing from equation (15b), 

X R = E a t r(t - IT), 

IfSs- 

the average over X R can be performed, we thus have 
f exp { - [y - r(t ) + X x ]X R /a 2 } dF(X R ) 

•'all Xk 

= II (exp \-[y - r(Q + X N ]a t r(t - lT)/a 2 }) al , (24a) 

where (g(x)) l means expectation of g(x). It has been shown* that the 
following inequality holds. 

(exp {a,z}) u( ^ exp (x 2 a 2 al /2) = exp {x 2 (2m - l)(2m + l)/6}. (24b) 

Substituting equation (24b) into (24a) we obtain 

f exp { - fo - r(to) + X s ]X R /a 2 \ dF(X R ) 

'all Xr 

^ exp {[y - r(t ) + X iY ] V R /2<r 4 } , (24c) 
* Appendix of Ref. 6. 
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where ol is given by equation (21c). Substituting equation (24c) into 
(23) we obtain the upper bound of equation (21a). 

It is interesting to note that the upper bound differs from the lower 
bound only through a modification of the noise power by the truncated 
terms. For a system with a peak distortion* less than unity, by taking 
the set S N large enough ol approaches zero, <j\ approaches a 2 , and the 
upper bound converges to the lower bound. Therefore, the exact error 
probability can be located within a small range. The computation 
time involved for large enough N is rather minimal when a digital 
computer is used as will be illustrated in Section V. 

4.2 Evaluation of P h and P u 

We have already shown in Ref. 6 that equations (17) and (21) can 
be expanded into an absolutely convergent series involving moments 
of the truncated intersymbol interference. 

The series expansion of equation (17) is 

P L = [(2m - l)/ro] erfc [-r(< o )/(2*<0] 

+ [(2m - l)/m] S [(2fc)!]- 1 (2(r 2 )- t (7r)- i exp (-r\t )/2a 2 ) 

A = l 

■H 2k -MQ/(2 i <r))M 2k , (25) 

where 

H 2 k-i is the Hermite polynomial, 

M 2k is the 2fcth moment of the random variable X^ . 

The series expansion of equation (21) is similar to equation (17), 
Pa = (<ri/o)|[(2m - l)/2m] erfc [-r(0/(2 i cr 1 )] 

+ [(2m - l)/m] E [(2fc)!]- 1 (2(7?)-*(7r)- i exp [-r\t )/2a 2 ] 

•H ik . 1 (r(to)/(2 i a l ))M 2k J- (26) 

The moments (M 2k ) can be obtained through the characteristic 



Normalized peak distortion (D p ) is defined as 

D P = t \r(to + nT)\/\r(t )\. 
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function of X x without the explicit evaluation of the distribution 
function. The recurrence formula for M 2k is 

M 2k = -Z^lI^a-oC-DM^'^m) 21 - l]/2i) \B 2i | 

•[Zi- IT) 2< ], (27) 



1*0 
IiSn 



where B 2i are the Bernoulli numbers. 

4.3 Truncation Error Bound of Series Expansion 

The error incurred by truncating the series of equation (25) at 
(n — 1) term is given by 

R n = [(2m - l)/m] f) (2fc!)- 1 (2o- 2 )-*(T)~ i exp [r 2 (Q/2a 2 ] 

k = n 

■H 2k -MQ/(2 h <r)]-M 2k . (28) 

Let 

X = max \ X N \ = (2m - 1) J2 \ r(t - IT) |. (29) 

MO 

It can be shown that the moments satisfy 

M 2k+2p ^ M 2k \ 2p , p = 0, 1, 2, • • • . (30) 

For (2k — 1) » x, the Hermite polynomials are upper bounded by 

I H ik . x (x) | ^ 2*-*[(2A - 3)!!] V2T^T exp [as»/2]. (31) 

Substituting equations (30) and (31) into equation (28) we obtain the 
following: 

| R n | £ [(2m - l)/m](2r)- J exp [-r 2 (Q/4a 2 ] 

■M 2n -C2<T 2 y n 2 (V)~\2k - l)-*(X 2 /2<r 2 )*-" 

lt=n 

^ [(2m - l)/m](27r)- J exp [-rU)/4<r 2 ] 
■M 2n -(2a 2 y n {z (k^\2k - l)-*(\ 2 /2<j 2 ) k - n 

+ (P!)" 1 (2p - l)^(\72«7 2 ) p - n (l - X72PCT 2 )- 1 } , (32) 
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Fig. 2 — The convergence of the series expansion method; (sin irt/T)/{trt/T) pulse, 
binary AM system, t = 0.05 T, SNR = 16 dB. The set S N includes the first 12 
sampling points around No- 



where p is an integer which is chosen to satisfy (\ 2 /2pa 2 ) < 1. Similar 
truncation error bounds can be obtained for P u . 



v. APPLICATION 

The error probability of a 2M-ary digital AM system with an ideal 
band-limiting pulse signal operating over an ideal channel is calculated 
by equations (25) and (26) to determine the convergence of the method. 
The received binary pulse is assumed to be 

r(t) = (sin *t/T)/(rt/T). (33) 

The system SNR is denned by 

SNR = (a 2 )r\0)/a 2 . (34) 

The convergence of the series expansion method is illustrated in Fig. 2. 
The system is binary with the sampling instant deviated by 0.05 T 
from its nominal sampling instant. The SNR is 16 dB. The set S lV 
includes 12 elements, i.e., I — ±1, ±2, ■ • • , ±6. It is observed that 
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Fig. 3 — Convergence of Pi as a function of number of terms in the series expansion; 
(sin *l/T)/(rl/T) pulse, 4-level digital AM system, t = 0.05 T, SNR = 23 dB. 
The set S N includes the first 12 sampling points around Mo- 



tile series converges after 3 or 4 terms. A similar example is given in 
Fig. 3 for a 4-level system with SNR = 23 dB. 

The convergence of the upper bound to the lower bound with in- 
creased size of S N is illustrated in Figs. 4, 5, and 6 for binary and 4-level 
systems respectively.* It is observed that the two bounds indeed 
merge together as N is increased. The upper and lower bounds on the 
error rate were calculated using a program written for the GE-Mark II 
time-sharing system. For the examples given here, computation time 
was less than a second. The change of N from 6 to 30 hardly had any 
effect on the computation time which indicates that one should start 
with S N sufficiently large such that a 2 R is small in comparison with <j\ 
probably of the order of 0.2 a 2 or smaller. Under this condition the 
upper and lower bound should be fairly close. As a comparison, the 
Chernoff bounds are also presented in Figs. 4, 5, and 6. 

The method given here can also be applied to the calculation of the 
error rate of a coherent phase-shift-keyed system. The error rate 
calculation of a two and four phase system can be reduced to the basic 



* The number of terms in the series expansion used to calculate the points in these 
figures are determined so that the truncation error is negligible. 
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Fig. 4 — Convergence of P u to P/,; ideal band-limited pulse, binary system, SNR 
16 dB, t = 0.05 T. 



formula, equation (10), 9 which then requires the determination of 
P[ Z aiKAi - IT) + n(to) ^ r(t )}. (35) 

The method described here can then be applied. Similar applications 
can also be found in the error calculation of co-channel interference. 
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Fig. 5 — Convergence of P u to P L ; ideal band-limited pulse, binary system, SNR 
16 dB, to = 0.1 T. 
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Fig. 6. — Convergence of P u to P L ; ideal band-limited pulse, 4-level system, SNR = 
16 dB, U = 0.05 T. 



VI. CONCLUSIONS 

We have presented a method to calculate the error rate of a coherent 
digital system subject to intersymbol interference and additive Gaussian 
noise. The error rate for a system with a peak distortion less than unity 
can be determined to arbitrary accuracy through the calculation of 
an upper bound and a lower bound of the error rate. The computation 
time involved (less than one second on the GE-Mark II time-sharing 
system) is many orders of magnitude shorter than the time required 
by the straightforward calculation of all the possible states. On the 
other hand, the results are generally much more accurate than the 
results obtained through the application of the Chernoff bound. 
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